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ERIC 



Introduction 



In 1993 the Commonwealth Department of Education Employment and Training 
commissioned the production of a detailed guide to evaluation for people working 
in the Adult Literacy and Basic Education (ALBE) field. The result was the 
production of a very comprehensive text, A Guide to Program Evaluation, written by 
Faye Lambert and John Owen in conjunction with Sharon Coates and Julie McQueen 
(DEET, 1994). 

Subsequently the National Staff Development Committee (NSDC) commissioned 
Rex Stoessiger and Roy Pugh of the Tasmanian Educational Consortium Ltd to adapt 
the Guide to become a professional development course. As a result this course. 
Professional Development for Program Evaluation, was written. It was trialled with 
ALBE personnel in a variety of Australian states and sections were then redrafted. 



1 COURSE AIMS 

This professional development program aims to: 

• improve participants' understanding of the range of approaches and methods 
used to achieve different evaluation purposes 

• provide practical training to assist participants, at any level within ALBE 
operations, to undertake evaluations that will lead to improved efficiency and 
effectiveness 

• point to ways of making evaluation techniques an integral part of program 
administration and implementation 

• demonstrate the application of evaluation processes within the ALBE field 
and, in particular, focus on the selection and use of performance indicators, 
reflecting the operational functions at national, state or territory, regional and 
provider level. 

(Adapted from A Guide to Program Evaluation, pp. 1-2) 

2 COURSE Structure 

The course is composed of six modules. Module 1 is a core module. 

Module 1 Introduction to Evaluation and an Overview of Evaluation Issues 

Module 2 Evaluation Approaches and Methodologies 

Module 3 Information Collection Alternatives 

Module 4 Collecting and Analysing Evaluative Information 

Module 5 Reporting and Ensuring Outcomes 

Module 6 Good Practice in Evaluation 
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See also Item 5 of this Introduction which outlines different ways of structuring 
the course to meet the needs of particular participants. 

In addition to the six modules, participants who are doing the full course will 
undertake an evaluation project at their own ALBE workplace, called the 
Participant's Evaluation Project. ^ 

PARTICIPANT'S EVALUATION PROJECT 

The overall course is designed so that participants can undertake an evaluation 
project as an action learning task, between sessions. This allows participants to 
use the ideas developed during the course in practical situations w^hile they are 
fresh in their minds. 

At the start of each new session participants have the opportunity to report on 
the progress of their project to date, allowing them to get assistance, and 
provHing useful insights for other course members on the practical consequences 
of conducting program evaluations. 

Participants are encouraged to attempt a small evaluation project which 
contributes to their normal work. The 'workplace' for the project may be a 
teaching/learning program in any ALBE context. This could, of course, include 
an ALBE program conducted at an industrial site or in a commercial enterprise. 

TOTAL NOMINAL TIME 

Each of the six modules takes about 3 hours of workshop time. 
The Participant's Evaluation Project will take about 15 hours extra. 

Total: 33 hours for the full course 



3 ENTRY REQUIREMENTS 

This program is for all personnel working in ALBE and related areas. 

♦ It is for people operating at national, state or territory, regional and local 
levels who need to undertake evaluations as part of their work. 

• It is for use by systems, departments, organisations and work groups. 

The course would be useful in a wide variety of areas to people who are involved 
with evaluation in general, and program evaluation in particular. The examples 
given largely relate to ALBE, but could easily be supplemented or replaced by 
examples from other areas as the basic approach applies to evaluation in any 
field. 

No background knowledge of evaluation is required lo start the course, but 
experience with evaluations will assist participants. 
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4 RECOGNITION OF PRIOR LEARNING 

Participants who wish to have prior learning recognised instead of completing 
this program may apply to the course coordinator of each module for assessment 
of current competence against the learning outcomes identified in the program. 

5 DELIVERY 

Modes of delivery 

The preferred approach in this professional development program is a highly 
interactive one. Adult learners have much to contribute to their own professional 
development and training. Good courses both respect and utilise the thinking of 
participants. 

As a result the course is a balance of relatively small amounts of presenter inputs 
with longer periods spent in organised group discussions. The use of a wide 
variety of group sizes is suggested, including pairs and small groups; and the 
whole group for brainstorming and discussions. 

Often participants report one of the major benefits of attending professional 
development sessions has been spending time with colleagues. Accordingly 
strategies such as group investigations with all participants reporting to new 
groups (see Additional Note 1.1 'Modified Jigsaw Movement') have been 
incorporated to ensure that participants have opportunities to work in a variety 
of groups as part of the course. 

RESTRUCTURING THE COURSE 

In Unit 1.4 participants undertake a self-evaluation of their prior learning about 
evaluation. The self-evaluation may suggest the need to re-structure or re-focus 
aspects of the course. Presenters will, ideally, be flexible enough to do this. For 
example, if all participants reveal a thorough knowledge of evaluation 
fundamentals then less time might be spent on Module 1 and more time devoted 
to subsequent modules. 

NUMBER OF PARTICIPANTS 

To enable effective small and whole group interaction, there should not be more 
than 20 participants. 

PRESENTERS' QUALIFICATIONS 

Presenters should: 

• be experience j in delivering professional development programs 

• have an excellent knowledge of the theory and application of evaluation 

• be up-to-date with the broad issues affecting the ALBE field. 



PROFESSIONAL DEVELOPMENT FOR PROGRAM EVALUATION 



PROGRAM FLEXIBILITY 
Full Course 

The six modules are designed to fit together to provide a comprehensive course 
in evaluation. Each module takes about half a day so the whole course could run 
as six half day sessions or as a three day course. In either of these formats 
participants can be asked to conduct their evaluation project as part of the course 
and report on its progress during subsequent sessions. 

Short Courses 

While designed to fit together, units can also be used independently. For 
example, Module 1 can stand alone as an introduction to evaluation. Similarly 
those with a reasonable understanding of evaluation issues who wish to find out 
more about a range of data collection alternatives could use Module 3 only. 

Modules 1 and 2 fit together well, as do Modules 3 and 4, and also Modules 5 and 
6. Any unit prerequisites are mentioned in the module overviews. 

Short courses, without time spent back at work, lose the option of undertaking 
the Participant's Evaluation Project as part of the course. 

Composite Courses 

Composite courses can be constructed by making careful selections from several 
modules. For example, a one day course for people with some background 
knowledge of evaluation might consist of the units shown in the following table. 



Unit 


Title 


1.8 


Planning an evaluation 


1.9 


The evaluation question 


1.10 


Evaluation for different purposes 


2.6 


Evaluation for program improvement 


2.7 


Evaluation for program justification and 




accountability 


3.4 


Information collection alternatives 


4.4 


Sampling 


5.5 


Ensuring the impact of evaluation findings 



Such combinations need to be made with care to ensure some connection 
between units. With this in mind it is also possible to negotiate with participants 
to select units which mecl their specific needs. 
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6 REQUIREMENTS FOR THE AWARD 



Learning outcomes, assessment criteria, conditions and assessment methods are 
listed for each module. 

Those who attend the full program and complete the assessment task to the 
required standard will receive a Statement of Attainment, provided that the 
course has been accredited in their state or territory. 

Otherwise the award will be a Statement of Achievement outlining the sections of 
the course completed. 



There have been no formal arrangements for recognition of the award. 

Participants may choose to present the Statement of Attainment, or Statement of 
Achievement, for advanced standing towards courses offered by tertiary and 
post-secondary institutions. 



The course is designed to correlate with competency statements developed by the 
Department of Education Employment and Training's International Literacy Year 
project, 'What Is a Competent Adult Basic Education Teacher?' In 1990 DEET set 
up this ILY project which was conducted by the University of Technology, 
Sydney. An interim report was published. What is a Competent ABE teacher? and 
in 1993, a final report. The ABE Profession and Competence: Promoting Best Practice. 
The UTS work was developed with cross sector input from all states and 
territories and provides the only framework available for ALBE personnel. 

The competency statements relevant to this course are found in chapter 4, section 
4.5, of The ABE Profession and Competence: Promoting Best Practice, p. 28. 



Unit 5: Evaluation of Programs 

Element 5.1 

Examines existing models of evaluation to adopt and 
incorporate appropriate evaluation strategies, and uses the 
evaluation of courses for revision, planning and development. 

Element 5.2 

Reports to relevant authorities on program effectiveness. 



7 Recognition of the award 



8 relationship to Competency Standards 
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9 RESOURCES 
THE MATERIALS 

Five types of materials are included in this package to assist with the delivery of 
this course. 

1 Presenter's Guidelines which provide a detailed description on how to present 
the course units. Presenter's Guidelines are identified by the letters PG 
highlighted in the header of each page. 

2 Overhead Projector Transparencies which are identified by the highlighted 
letters OHP. 

3 Handouts for participants, with the letter H highlighted. 

4 Additional Notes which provide advice to the presenter about aspects of the 
course and its presentation and have the letter AN highhghted. 

5 Case Studies which are placed in an appendix. 

Three published evaluations of ALBE projects have been re-written in an 
easily accessible style to provide case studies of how evaluation projects have 
been conducted. The case studies are referred to at key points in the 
Presenters' Guidelines, but can also be copied and distributed to participants 
as supplementary readings. 

A Guide to Program Evaluation 

The above materials, except for the case studies, are adapted from A Guide to 
Program Evaluation, referred to throughout this document as 'the Guide'. 

One copy of ^4 Guide to Program Evaluation is supplied with each of these 
packages. 

The Presenter's Guidelines refer to the pages of the Guide that the presenter will 
need to copy. It is essential that presenters study the Guide thoroughly and use it 
alongside this professional development course. The Guide is much more 
comprehensive than this document, as only some parts of the Guide could be 
condensed into a course of reasonable length. Hence the Guide is also 
recommended for further reading, along with additional references for 
participants who wish to continue learning about evaluation. 

The Commonwealth of Australia copyright specifications for A Guide to Program 
Evaluation allow it to be 'reproduced in whole or in part for study or training 
purposes subject to inclusion of an acknowledgment of the source; it is not to be 
copied for commercial use or sale' (The Guide, p. i). Presenters must respect these 
conditions. 

ERIC 6 vi) NSDC/Commonwealtti of Australia 1995 
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10 ASSESSMENT 

There are two assessment methods for participants doing the full course. 

1 They will be assessed according to lite level of their participation in the 
workshop activities. 

2 They will conduct a Participant's Evaluation Project which includes: 

- developing an evaluation plan for a program at their own workplace 

- collecting the information required by the evaluation plan 

- processing the information collected 

- reporting on the evaluation findings in some suitable format. 

In Module 6 an interim report will be given to the group by each participant 
doing an evaluation project. Within one month of the program a final project 
report should be sent to the coordinator who will be responsible for assessing 
whether participants have satisfactorily completed their evaluation project. 

The coordinator and/or presenter will assess whether participants have achieved 
the learning outcomes for this course. 

The coordinator and/ or presenter will ensure that a Statement of Attainment (or 
Statement of Achievement, if appropriate) is sent to each successful participant 
and will also handle requests for recognition of prior learning. 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Course Overview 



1 Introduction 



2 Approaches and methodologies 

3 Information collection 

4 Collecting and analysing information 

5 Reporting and ensuring outcomes 



6 Good practice in evaluation 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Agenda - Module 1 

1 Welcome and introductions 

2 Overview of the course 

3 Agenda for the first module 

4 Recognition of prior learning 

5 A definition of evaluation 

6 The components of an evaluation 

7 The importance of context 

8 Planning an evaluation 

9 The evaluation question 

10 Key Aspect 1: 

Evaluation for different purposes 

1 1 Planning an evaluation project 

12 Session evaluation and close 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



A Definition of 
Evaluation 

Evaluation is: 

the collection and 

the analysis of information 

in order to 

facilitate 

informed decision making 

from The Guide, p. 1 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



The Stages of an Evaluation 



1 Planning 

2 Data collection 

3 Data processing 

4 Making judgments 



5 Reporting 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Modified Jigs a w Movement 




EKpert 



Reporting 
group 1 



Reporting 
group 2 



Reporting 
group 3 



Reporting 
group 4 



Adapted from a diagram in Bennett, B., Rolheiser- Bennett, C. & Stevahn, L. 1991, 

Cooperative Learning: Where keart meets mind, 
Educational Connections, Toronto, Ontario, p. 219 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Planning an Evaluation 



What is to be evaluated? 
Why? 

Who is involved? 

Who is the evaluation for? 

How will the program be evaluated? 

When? 

Who by? 



. , Transparency 1.6 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



The Evaluation Question 



An ALBE project management team 
wants to include 'everything' in the 
evaluation, but resources are limited. 

How would you go about encouraging 
them to focus on particular evaluation 
questions? 



45 



Transparency 1.7 



© NSDC/Commonwealth of Australia 1995 35 



PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Focus Questions 



In what ways are different groups (e.g. 
funding agencies, managers and 
practitioners) interested in different 
purposes for an evaluation? 

What might be the consequences of 
these differences? 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



The purpose of the course 



The course is designed to: 



improve your understanding of the range of approaches 
and methods used to achieve different evaluation 
purposes 

provide practical advice that will assist you, at any level 
within ALBE operations, to undertake evaluations that will 
lead to improved efficiency and effectiveness 

point to ways of making evaluation techniques an 
integral part of program administration and 
implementation 

demonstrate the application of evaluation processes 
within the field of adult literacy and basic education, and in 
particular, focus on the selection and use of performance 
indicators, reflecting the operational functions at national, 
state/territory, regional and provider level. 



Adapted from The Guide, pp. 1-2 



ERIC 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Definitions of Evaluation Terms 



Appraisal is evaluating an individual's performance or vyork. 

Assessment is collecting information in order to make judgments about 
learning, competency or attainment. 

Effectiveness is the extent to which a program is satisfying the purpose 
for which it was established. 

Efficiency is the economic use of inputs to achieve the program's 
outputs. 

Evaluation is the collection and the analysis of information in order to 
facilitate informed decision making. 

Performance Indicators are simple statistics designed to give summary 
information which allows managers and others to make judgments about 
the effectiveness of programs. They are usually expressed as an index, 
rate or other numerical figure. 

Performance Information is any information (both quantitative and 
qualitative) which allows judgments to be made about the achievements 
of a program. 

Reliability of an approach is its ability to obtain the same result on 
repeated occasions. 

Self-evaluation is the evaluation of a program by the people responsible 
for planning and implementing it. 

Stakeholders are people or groups who are significantly affected by the 
actions of a program or who impact on the program in important ways. 

Validity of an approach is its ability to measure what it is intended to 
measure. 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Evaluation Form 1 



Please fill In the evaluation form to help Improve subsequent sessions. 
1 What helped your learning In the session? 



2 What suggestions do you have for Improving subsequent sessions? 



Please rate the following aspects of the session on a 1 to 4 scale 
with 1 meaning very good and 4 meaning very poor. 







1 


2 


3 


4 


3 


The session content 


□ 


□ 


□ 


□ 


4 


The style of presentation 


0 


□ 


□ 


□ 


5 


Depth of material covered 


□ 


□ 


□ 


□ 


6 


The type of activities 


□ 


0 


n 


□ 


7 


The support materials 


□ 


□ 


□ 


□ 


8 


Interaction with other participants 


□ 


□ 


□ 


□ 


9 


Getting what you came for 


□ 


□ 


□ 


□ 



10 Any other comments? 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Additional Note 1.1 
Modified Jigsaw Movement 



A common strategy used in this course is a group investigation in which 
participants are grouped in different ways in a movement based on the Jigsav/ 
structure used in the cooperative learning literature (Bennett, B,, Rolheiser- 
Bennett, C. & Stevahn, L. 1991, Cooperative Learning: When Heart Meets Mind, 
Educational Connections, Toronto, p. 219). 

The structure used here is a two step process. 

1 Divide participants into groups of roughly equal numbers. These are the 
expert groups. In these groups participants investigate a particular topic and 
prepare to report, individually, to new groups who have not studied this 
particular topic. 

2 Move expert group members to new groups for reporting on their findings. 
Move the first person in expert group 1 to reporting group 1 the second 
person to reporting group 2, the third person to reporting group 3 and so on. 
Similarly Ihe first person in expert group 2 moves to reporting group 1 the 
second person to reporting group 2 and so on. 

The movement is shown in the diagram. 



ERLC 




Ewpert 



Reporting 
group 1 



Reporting 
group 2 



Reporting 
group 3 



Reporting 
group 4 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



AM 



Modified Jigsaw Movement 
Additional Note 1.1 (continued) 



The movement is designed so that each reporting group has one member of each 
of the expert groups. 

In the reporting groups each participant will report on the information collected 
in the expert group. In this way everyone has a turn as a reporter and everyone 
hears from all the other groups. It is important to let participants know^ at the 
start that they will all end up being reporters so they can keep good notes. 



Note: If there are uneven numbers two people from one expert group may go to 
the same reporting group and share the reporting. Alternatively the leader may 
join a group to make up the numbers. 

If there are four expert groups containing five members each then, after the 
movement, there will be five reporting groups containing four participants each. 
Similarly if there are three expert groups containing four members each the 
result will be four reporting groups containing three members each. 



While the movement may appear daunting at first sight it is well worth 
persevering with because it has been shown in the trials to be a highlight with 
participants. 
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FOR PROGRAM EVALUATION 



Additional Note 1.2 
An Example of the Importance of Context 



A major project defined 'rurality' by correlating statistical indices against 
common sense definitions of rurality for several Australian areas. 

The consequence was that the index of rurality was defined to be based on: 

1 The number of vehicles per household: the more vehicles the more rural. 

2 The proportion of households with TV sets: the less TVs the more rural. 

3 The percentage of the work force in rural industries. 



In the national context this definition works very well and provides a useful 
index of rurality. But when applied in Tasmania, a rural town — New Norfolk — 
became the most urban locality in the state! 

New Norfolk is a mill town with very high proportions of the work force in 
manufacturing industry and very low proportions in rural industries. It has 
perfect TV reception and very high set ownership. Being a mill town with bus 
travel provided to the plant one car families are the norm. Yet in the Tasmanian 
context New Norfolk behaves as a rural centre in most ways. 

Consequently the local context can make an index or procedure developed for a 
different, or broader, context quite unsatisfactory and could lead to invalid results 
being obtained. 
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Additional Note 1.3 
Suggestions for Focusing on Particular 
Evaluation Questions 



Here is the task: 



An ALBE project management team wants to include 'everything' in the 
evaluation, but resources are limited. How would you go about 
encouraging them to focus on particular evaluation questions? 



In response to the task participants might make some of the suggestions listed 
below. 

1 Make a list of possible evaluation questions then prioritise them. 

2 Ask a small number of key stakeholders to submit one question each. 

3 Exclude all questions that are peripheral to the stated purpose of the 
evaluation. 

4 Work out an estimated cost to tackle all the evaluation questions and 
compare with the funds available. 

5 Ask for twice as much money and staff to conduct the evaluation as budgeted 
for. 

6 Conduct a survey with stakeholders (e.g. using a nominal group technique — 
see Module 3) to determine which questions are essential for the evaluation. 

( 

Individuals will probably suggest a wide range of ideas. 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Agenda - Module 2 

1 Welcome and agenda 

2 Report on work to date in planning an 
evaluation 

3 Key Aspect 2: 
Progressive focusing 

4 Evaluation for program development 

5 Evaluation for program clarification 

6 Evaluation for program improvement 

7 Evaluation for program justification 
and accountability 

8 Participants' Evaluation Projects 

9 Session evaluation and close 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Evaluation FOR 
Program Development 



PURPOSES OF 
EVALUATION 


PROGRAM DEVELOPMENT 

to make decisions about how 
best to develop a program to 
meet the needs, e.g. training 
needs analysis 


FOCUS 


context 


TIMING 


before 


STAGE OF 
PROGRAM 
DEVELOPMENT 


none 


TYPICAL 
QUESTIONS 


• What is the ideal or desired 
state of affairs? 

• What is the actual state of 
affairs? 

• Are there discrepancies 
between the desired and the 
actual? 

• What are the reasons for the 
discrepancies or needs? 


TYPICAL 
DECISIONS 


• Should the development of a 
program go ahead? 

• What needs should be given 
priority for action through an 
inten/ention or program? 


ALBE 

EXAMPLES 


The Guide 
pp. 19-20, pp. 22-23 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Training Needs Analysis 



Combining the LTA and the LSA to provide the LTNA 



Literacy Task Analysis Literacy Skills Audit 

literacy tasks required literacy skills possessed 

on the job or lacking 



Literacy Training Needs Analysis 

Comparison of Literacy Training Needs 
and Literacy Skills Audit 
to determine training needs 

^l^ 

Curriculum Response 



from Workforce Literacy Training Package, p. 122 
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FOR PROGRAM EVALUATION 



Evaluation FOR 
Program Clarification 



This form of evaluation is concerned with 
clarifying the underlying logic of a 
program. 'Logic' here is taken to mean 'a 
conceptual view of how a given program 
operates' . 

It involves describing how a program is 
supposed to work, identifying cause-and- 
effect relationships and, in doing so, 
provides a rationale for the program as an 
intervention. 

The Guide, p. 25 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Approaches to 
Program Improvement 

♦ implementation studies 

♦ refining developing programs 

♦ site-level evaluation and action 
research 

♦ continuous improvement 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Evaluation FOR 
Program Justification 
AND Accountability 



A major purpose of evaluation is to 
account for the resources expended on the 
program under review. 



Generally, accountability evaluation 
fulfils the need for someone directly 
associated with the program to report to 
somebody else, e.g. the funding agency. 

In this scenario the manager justifies the 
funds and other resources used to provide 
the program. 

Adapted from The Guide, p. 45 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Monitoring Performance 



Monitoring the performance of 
established programs is done to: 

♦ provide information for the 
management of programs and 

♦ provide the basis for external 
accountability. 

It is an integral part of the management of 
programs. Managers need to: 

♦ have information that will help them 
plan and control service delivery and 

♦ account for their program and 
organisational responsibilities. 

Adapted from The Guide, p. 45 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Impact Evaluation 



Impact evaluation also focuses on the 
delivery and the outcomes of well 
established programs, but the emphasis is 
on the assessment of outcomes. The 
primary purpose is to justify the worth of 
the program and to account for program 
expenditure. 

These evaluations are usually conducted 
towards the end of a program. 



Adapted from The Guide, p. 45 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Project Management Cycle 



Policy/Decision-m^ing 



Monitoring 




Implementation 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Performance Indicators 



A performance indicator is a simple 
statistic designed to give summary 
information which allows managers and 
others to make judgments about the 
effectiveness of programs. 

Information is usually expressed as an 
index, rate or other numerical figure. 
Performance indicators are, in effect, 
substitutes for things that cannot be 
assessed directly. 

For example, a medical symptom such as 
'a raised temperature' is an indicator of 
illness. 

NSW Public Service Board, 1986 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Impact Evaluation Questions 



What are the 'real goals' of the program 
and to what extent have they been 
attained? 

What are the intended and unintended 
outcomes of the program? 

How do differences in implementing the 
program affect the outcomes? 

Does the program meet identifiable 
needs? 

Vie Guide, p. 53 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Continuous Improvement 



Total Quality Management has popularised processes of continuous 
improvement suggested by W Edwards Deming and developed as 
Kaizen by Imai in Japan. 

Continuous improvement is a small step improvement process. It usually 
involves a small group evaluating a process which they control to 
determine how it could be improved. 

It involves a cyclic, four step process of planning, doing, checking and 
acting or standardising. 



/ HCT 


PLAN \ 


\ CHECK 


DO / 



1 Plan 

Identify the process or problem of interest. Collect information on the 
problem. Use the information collected to plan improvement. Determine 
what is to be done, who will do what, when, and ensure the required 
resources are available. 



2 Do 

Carry out the plan. Ensure stakeholders know about it. Provide any 
required training. 
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FOR PROGRAM EVALUATION 



CONTINUOUS IMOROVEMENT (CONTINUED) 



3 Check (or monitor) 

Collect the necessary information to monitor what happens against the 
plan to ensure improvement is occurring. Discuss the results in the 
group and identify reasons for what is observed. Reflect on the findings 
and develop recommendations from the experience. 



4 Act (standardise tlie improved performance) 

If the checking stage confirms that improvements have resulted then 
make the new method the standard. Advise all who need to know about 
it and incorporate it in training for new staff. Establish further monitoring 
procedures to ensure the gains are maintained. 

If the improvements are not achieved then review the data and decide 
whether to abandon the approach or to re-plan it in a different way and 
repeat the cycle. 



Continuous improvement is very like other improvement cycles, for 
example action research involves a cycle of plan, implement, monitor 
and reflect. 

Continuous improvement can be contrasted with the focus on innovative 
solutions popular in Australia. 



ERIC 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Evaluation Form 2 



Please give us your views of the following aspects of the session. 







V. good 


good 


fair 


poor 


1 


The course content 


□ 


□ 


□ 


□ 


2 


The style of presentation 


□ 


□ 


□ 


□ 


q 
o 


1 ne venue 


n 


□ 

LJ 


□ 

LJ 


□ 

1 — 1 


4 


The organisation 


□ 


□ 


□ 


□ 


5 


The support materials 


□ 


□ 


□ 


□ 


6 


Group activities 


□ 


□ 


□ 


□ 


7 


Individual presentations 


□ 


□ 


□ 


□ 


8 


Getting what you came for 


□ 


□ 


□ 


□ 



Other comments? 



9j 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Agenda - Module 3 

1 Welcome and outline of the agenda for 
the workshop 

2 Report on work to date 

3 Key Aspect 3: 
Making judgments 

4 Collecting information 

5 Information collection alternatives 

6 Information collection — 
questionnaires 

7 Participants' Evaluation Project 

8 Session evaluation and close 

^ Transparency 3.1 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Referencing Evaluation 
Judgments and Decisions 

norm referencing 

competency or criterion referencing 
standards referencing 



Transparency 3.2 



1V3 ' 
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r^plBj (UiU) PROFESSIONAL DEVELOPMENT 

FOR PROGRAM EVALUATION 



Collecting Information to 
Improve A Program 



If you wanted to determine how to 
improve a prison literacy program 
where would you get the information 
and how would you collect it? 



Transparency 3.3 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Information Collection 
Alternatives 



1 Written responses, diaries etc. 

2 Checklists or inventories 

3 Rating scales, Delphi technique 

4 Ranking scales and semantic 
differential scales 

5 Written accounts 

6 Observation forms 

7 Face to face interviews and telephone 
interviews 

8 Group interactions, focus groups etc. 

9 Search conferences 

10 Audio and video taping 

1 1 Photography and unobtrusive 
techniques 

12 Existing records 
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FOR PROGRAM EVALUATION 



Evaluation Form 3 



Please make a note of your views about various aspects of the course as the 
session proceeds. 

1 Things you liked 



2 Things that could be improved 



3 Your comments or suggestions 



Handout 3.1 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Additional Note 3, 1 
Referencing Evaluation Judgments 
AND Decisions 



Evaluation involves making judgments about the program or activity being 
evaluated. Anybody can make judgments, and in most situations different people 
will make different judgments. Whose judgments will we accept? 

For these reasons it is necessary to agree on some process which will reference, or 
validate, the judgments made as part of the evaluation. This is common in the ALBE 
field when assessing student performance: some sort of referencing system may be 
used to help ensure that valid and fair assessments are made. 

The common referencing systems used in education are: 
norm referencing 

competency or criterion referencing 
standards referencing 



Norm referencing is a comparison to the performance of others, ideally to a large 
population of others whose performances are distributed on a normal curve. Tertiary 
entrance assessments are usually norm referenced. 

Competency or criterion referencing makes judgments based on the competencies 
required to perform a particular task or the criteria for demonstrating a particular 
outcome. The judgments are referenced against the work or task. 

Standards referencing makes judgments against some agreed standard of 
performance. The standards are usually somewhat arbitrary but are either set by 
experts in the field or by agreement between most practitioners. The Australian 
Curriculum Profiles are standards for teachers to use in assessing student 
performance. 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Agenda - Module 4 

i — 

1 Welcome and outline of the agenda for 
the workshop 

2 Report on work to date 

3 Reviewing information collection 
techniques 

4 Pilot studies 

5 Sampling 

6 Key Aspect 4: 
Reliability and validity 

7 Information analysis 

8 Participants' Evaluation Projects 

9 Session evaluation and close 

Transparency 4.1 

14b 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Pilot Studies 



A pilot study is a small scale study 
designed to test the effectiveness and 
efficiency of selected data collection 
techniques and instruments. This 
involves trying out the instruments under 
conditions similar to those expected in the 
evaluation, and so it should include a 
range of people similar to those who will 
participate in the evaluation. 



The Guide, p. 117 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Sampling 



Sampling can provide an efficient and 
accurate way of obtaining information 
about a large number of cases. Just how 
efficient and accurate depends on the type 
of sample used, the size of the sample and 
the method of collecting data from the 
sample. In the end decisions about 
samples will be a compromise between 
cost, accuracy, the nature of the research 
problem and the art of the possible. 

The Guide, p. 120 
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FOR PROGRAM EVALUATION 



Reliability AND Validity 



An evaluation approach is considered 
reliable if the same result is obtained on 
repeated occasions. 

An evaluation is considered valid if it 
measures what it is intended to measure. 

An evaluation approach should, of course, 
be fair to all those involved. 



149 
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FOR PROGRAM EVALUATION 



Quantitative AND Qualitative 
Data Handling 



Quantitative data handling: 

'What can you do with numerical data 
about two different ALBE programs to 
allow the data to convey meaning to 
others?' 



Qualitative data handling: 

'What could you do with descriptive 
and interview data about two different 
ALBE programs to allow the data to 
convey meaning to others?' 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Evaluation Form 4 



Please rate each of the statements below using the numbers 1 to 4 on the following 
scale: 

Strongly agree Agree Disagree Strongly disagree 

1 2 3 4 



1 Clear information was conveyed. 

2 i could ask the questions I wished to. 

3 I had the opportunity to work with a variety of people. 

4 I couldn't understand the materials. 

5 The session was well focused. 

6 I enjoyed learning about evaluation. 

7 The activities were varied. 

8 We were rushed for time. 

9 The presenter was a good role model. 

10 I'm looking forward to the next session. 



Any other comments? 




Handout 4.3 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



A GENOA - Module 5 

1 Welcome and agenda for the 
workshop 

2 Report on work to date 

3 Evaluation as proof and evaluation as 
investigation 

4 Determining audience needs 

5 Reporting options 

6 Report content 

7 Key aspect 5: Ensuring the impact of 
evaluation findings 

8 Reporting on Participants' Evaluation 
Projects 

9 Session evaluation and close 



Transparency 5.1 




OHP 




PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Focus Question - 

E VALUATION AS PrOOF OR 

Evaluation AS Investigation? 



In ALBE settings — 

♦ when is: 

'evaluation as proof 

most likely to be appropriate? 

♦ and when is: 

'evaluation as investigation' 
most likely to be appropriate? 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Different A udiences 



Different audiences often require 
different information. 



The Guide, p. 133 



Transparency 5.3 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Roles 



Problem owner 




Facilitator Timekeeper 



Focused Problem Solving Roles: 
A Problem owner 
B Questioner 
C Questioner 

D Recorder, facilitator and reporter 

E Timekeeper, question monitor and 
process observer 

Transparency 5.4 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Focus Question - 
Problem Solving 



How would you ensure that program 
decision makers understand the results 
of evaluations and act on them? 

How could we incorporate effective 
change processes into an evaluation right 
from the start? 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Evaluation Form 5 



1 . Please write down at least seven adjectives that describe your feelings about the 
session. 



2 Circle the three adjectives (above) that you feel most strongly about. 



3 Use the circled adjectives in one or more sentences to provide constructive 
feedback to the presenter and course designers about the session. 



4 Any other comments? 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 




Additional Note 5. 1 
Focused Problem Solving Activity 



The activity is a group problem solving process for groups of four or five people. 



Firstly a problem is ider\tified and formulated as a question. 

Next form groups of four or five participants and assign one person to be the 
problem owner (A), two to be questioners (B and C), one to be the facilitator (D) and 
a fifth (if available) to be timekeeper and monitor (E). 

Group members offer to take on (or are assigned) the roles. 

A The problem owner 

The person who will adopt the problem as their own. 

A can be any member of the group, not necessarily the one whose problem it is in 
the first place. This can offer a degree of objectivity to the problem solving 




Problem owner 



Questioner ( B 



Questioner 




process. 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Additional Note 5. 1 
Focused Problem Solving (continued) 



BandC 

The questioners 

Two members, B and C, take on the role of asking questions only. They must 
strictly stick to the procedure of asking only one question at a time, in turn, and 
not to enter into any discussion; 

About seven minutes are allocated to each of three question stages: 

stage 1: fact seeking questions only, 

stage 2: opinion seeking questions only, and 

stage 3: solution directed questions only. 

D Recorder, facilitator and reporter 

D records all the possible solutions that emerge from the questioning stage on a 
vertical surface such as a white board, if possible, for all the group to see. 

Once question time is over D facilitates a ten minute debriefing stage. All 
members share their thoughts feelings and opinions about the process. D then 
encourages further discussion of solutions and prepares the group's solutions 
ready to report to others. 

E Timekeeper, question monitor and process observer 

E is the timekeeper and monitors the nature -^f the questions asked. If they are 
not strictly of the type required £ asks the questioner to rephrase the question 
appropriately. 

E also takes the role of observer of the process in order to identify whether the 
initial problem question shifts during the activity. 



Group facilitators report on solutions to the whole group. 

If different ideas emerge from the different groups further discussion may be 

encouraged to see if a consensus emerges. 
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PROFESSIONAL DEVELOPMENT 
FOR PROGRAM EVALUATION 



Agenda - Module 6 

1 Welcome and agenda for the 
workshop 

2 Report on work to date 

3 ALBE evaluation practice 

4 Standards for evaluations 

5 Key Aspect 6: 

The rights of human subjects 

6 Managing evaluations and using 
consultants 

7 Course evaluation and close 
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Questions to help your partner reflect on 
his or her work: 

♦ What did you do well? 

♦ What did you learn? 

♦ What would you do differently 
next time? 
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Assumptions about 
Acceptable Evaluation 
Practice for ALBE 



ALBE evaluations must: 

♦ be 'owned' by the field 

♦ identify legitimate stakeholders 
(including the Commonwealth) and 
include them in decision making 
about evaluation 

♦ make evaluation processes simple 
and accessible to stakeholders 

♦ focus on critical areas of adult basic 
education, and 

♦ be cost effective. ne Guide, p. 139 
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Priority Areas for 
Evaluation Standards 



The standards for evaluation of 
educational programs are grouped in the 
following four areas, in order of priority: 

♦ utility 

♦ feasibility 

♦ propriety 

♦ accuracy. 

The Guide, p. 141 
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Discussion Question - 
The Ideal Consultant 



What criteria would you use for 
choosing a consultant for an 
evaluation? 

What procedures would you put in 
place to help manage the evaluation? 
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Evaluation Form 6 



Please give your views of the following aspects of the overall course. 



V. good good fair poor 



1 


The course content 


□ 


□ 


f 1 

□ 


1 1 

□ 


2 


The style of presentation 


□ 


□ 


□ 


□ 


3 


The support materials 


□ 


□ 


□ 


□ 


4 


Group activities 


□ 


□ 


□ 


□ 


5 


Getting what you wanted from the course 


□ 


□ 


□ 


0 


6 


The usefulness of the course to your work 


□ 


□ 


□ 


□ 



7 Can you suggest any improvements to the arrangement of the modules that 
make up the course? 



8 Are there any additional materials or topics that you would like included in the 
course, or any you would like excluded? 



9 List three things you have learnt from the course. 



10 Other comments? (Please continue over the page if you wish.) 
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Suggested Readings 



Bennett, B., Rolheiser-Bennett, C. & Stevahn, L. 1991, Cooperative Learning: 
Where Heart Meets Mind, Interactive Resource Book, Toronto. 

Bomford, P. 1992, Workplace Basic Education Evaluation Report, DEIRT, Hobart. 

Grant, A. 1987, Opportunity to Do Brilliantly: TAPE and the challenge of adult 
literacy provision in Australia, AGPS, Canberra. 

Impact of a National Framexvork for Professional Development of Adult Literacy and 
Basic Education Personnel 1994, National Staff Development Committee, 
Chadstone, Vic. 

Lambert, F. & Owen, J. (in conjunction with S. Coates & J. McQueen) 1994, A 
Guide to Program Evaluation, DEET, Canberra. 
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Appendix 

Case Studies 

OF 

Adult Literacy and Basic Education 

Evaluations 
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Introduction 



Three ALBE evaluations have been chosen as support materials to the course. 
Professional Development for Program Evaluation: Evaluation for Adult Literacy and Basic 
Education Programs. 

The evaluation reports have been condensed and reorganised as mini case studies 
showing how others have gone about evaluations, the methodologies they have 
adopted and the results and recommendations flowing from the studies. They range 
from national to local programs. 

The three examples were chosen because, collectively, they demonstrate various 
significant features: 

• evaluation used for a variety of different purposes 

• processes which utilise different methodologies 

• the use of uncomplicated data gathering and processing techniques. 

Each one throws up interesting evaluation issues and should be relevant to a variety 
of ALBE staff. 




Selected evaluations 

The three evaluations selected were: 

1 The Workplace Basic Education Evaluation conducted in 1991 by Pat 
Bomford of Department of Employment, Industrial Relations and Training 
(DEIRT now DIRVET), in Tasmania. 

2 An independent evaluation of the Impact of a National Framework for 
Professional Development of Adult Literacy and Basic Education 
Personnel, conducted by Price Waterhouse Urwick in 1993 for the National 
Staff Development Committee for Vocational Education and Training 
(NSDC), then TAPE NSDC. 

3 A study. Opportunity to do Brilliantly: TAPE and the challenge of adult 
literacy provision in Australia, conducted in 1984 by Audrey Grant for the 
Commonwealth Government and containing evaluations of two innovative 
adult literacy projects in Victoria. 
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HEADINGS FOR THE CASE STUDIES 

Module 1 raises discussion of some of the components of an evaluation, the 
evaluation context, some headings for planning an evaluation, the evaluation 
question and the different purposes of evaluations. It includes discussions of 
stakeholders, audiences and why different evaluation purposes require different 
methodologies. 

Accordingly, the evaluations included here as case studies have been summarised 
under the headings: 

What is to be evaluated? 
Context 

The evaluation question 
Purposes of the evaluation 
Who is involved? 
Audiences 
Methodology 

The following headings were also included: 

Findings of the evaluation 
Recommendations 

In addition, any issues arising from the case studies are discussed. 



210 



© NSDC/Commonwealth of Australia 1995 1 91 



PROFBSSIONAL DEVELOPMENT FOR PROGRAM EVALUATION 



Case Study 1 

Workplace Basic Education Evaluation 

Report 



WHAT IS BEING EVALUATED? 

All 22 Workplace Basic Education (WPBE) courses conducted in Tasmania during 
1991 and seven of the 26 courses completed during 1992. (Bomford, P. 1992, 
Workplace Basic Education Evaluation Report, DEIRT, Hobart and Bomford, P.' 1992, 
Evaluation Research. Questionnaires and key documents from the evaluation of Workplace 
Basic Education Tasmania, DEIRT, Hobart.) 



Context 

The first workplace basic education courses started as a pilot program in Tasmania 
m 1988. The pilot was considered an outstanding success and the program was 
developed with a total of 36 courses conducted between 1988 and 1991. Courses 
were also run for Skillshare and the Commonwealth Rehabilitation Service, but these 
were not included in the evaluation which was restricted to courses conducted in 
workplaces. 

Courses were initially government-funded, but evolved to become fee-for-service 
courses by the time of the evaluation. 

From 1988 to 1991 WPBE in Tasmania was administered by a central unit in Hobart. 
In January 1992, WPBE was regionnlised; the north-west, northern and southern 
regions of adult education became responsible for WPBE in their own regions. 



THE Evaluation Question 

The evaluation was designed to: 

measure the effect of WPBE on course participants' work roles and personal 
development 

ERIC 21 i 
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gauge the impact of WPBE on the work unit as a result of the individual 
undertaking the course 

identify benefits and problems from the point of view of the service user and 
service provider 

calculate the cost to the service user and service provider. 



PURPOSE OF THE EVALUATION 

The evaluation was for program justification and accountability purposes (see Unit 
2.7 of Module 2). It includes both 

♦ accountability aspects such as costs, benefits and effectiveness and 

♦ impact issues such as evaluating the effect and impact of the courses. 



WHO IS INVOLVED? 

The stakeholders mentioned in the evaluation report were course tutors, managers 
of ALBE programs, managers and supervisors of businesses sponsoring courses, 
union representatives and course participants. 

The evaluation was conducted by Pat Bomford, an in-house evaluator with the 
Evaluation and Research section of DEIRT (now DIRVET) in Tasmania. 



AUDIENCES 

Public service managers, industry, funding bodies. 



ERIC 



METHODOLOGY 

Minutes of the planning groups established at each workplace to advise on WPBE 
courses were examined to identify key issues. 

Structured interviews were held with course participants, supervisors, union 
representatives and employer representatives (usually training officers). Some 
interviews were in person, some by phone. A detailed questionnaire was constructed 
for the interviews of each group and trialled in a separate work plate. The interviews 
often deviated from the questionnaire and took the form of open discussion. 

Course tutors prepared a report at the end of each course and a discussion group 
was held with six of the tutors. 

212 

© NSDC/Commonwealth of Australia 1995 1 93 



PROFESSiONAL DEVELOPMENT FOR PROGRAM EVALUATION 



FINDINGS 

• Benefits from the WPBE courses for participants included: 

participants' personal development 

improvements in practical skills such as report writing and the use of 

calculators 
gains in confidence and 
improved communication skills. 

• Teamwork in the workplace also improved. 

• Twenty nine of the 52 participants interviewed said they learnt what they hoped 
to learn from the course while another 17 learnt 'to some extent'. Forty on.e of the 
52 participants said that the course had helped them in life outside work. 

• Employers were happy with the achievements of individuals on the courses. 
However, in workplaces where only a small number of participants completed 
courses employers found it difficult to justify the costs. Attendance problems 
occurred in five of the fourteen workplaces. Overtime, family commitments and 
shiftwork contributed to these. 

• Employers could optimise course benefits and attendance by running courses 
totally within work time and cultivating a workplace culture that encourages 
training. 

• Additional funding is needed to support WPBE. Employers were unwilling to 
spend more on courses. The author proposes that governments should share the 
financial responsibility with industry. 

• Only 14% of course participants were female. Family commitments made women 
reluctant to participate in courses run outside of work time. 

• WPBE needs strong and skilled marketing in the community and inside industry. 

• There is a need for good communication between the regional structure for 
WPBE. 

• The changes in industry, training and accreditation of training have important 
implications for WPBE. 



Recommendations 

A number of recommendations were made lo WPBE in Tasmania, but they were not 
outlined in the report. 
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ISSUES ARISING 

Focusing occurred at several stages. The questionnaires used in the interviews were 
refined in the light of experience. The plan to interview all 195 course participants 
had to be abandoned and interviews were only conducted with 52 participants. The 
plan to follow highly structured questionnaires in the interviews, which were even 
trialled before use, had to be abandoned when open discussion proved more useful 
in the interview situations. (Unit 2.3 of Module 2 has a discussion of progressive 
focusing.) 

Impact or accountability evaluations (Unit 2.7 of Module 2) which try to compare 
costs and benefits are often difficult in social contexts. In this evaluation the financial 
costs were carefully calculated in dollar terms. The benefits were such things as 
personal development and report writing ability. It is very difficult to compare these 
and the report, wisely, makes no real attempt to do so. 

Evaluations often lead to the identification of unexpected outcomes and issues; in 
this case, the low involvement of women in training programs. 

The evaluator may have attempted to ensure the impact of the evaluation by 
directly, and privately, communicating the recommendations to management rather 
than indirectly through the report (see Unit 2.7 of Module 5). An in-house evaluator 
can sometimes make stronger recommendations privately to management than can 
be made in a public report. 
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Case Study 2 

Impact of a National Framework for 
Professional Development 

WHAT IS BEING EVALUATED? 

In 1991 the TAPE National Staff Development Committee (now NSDC) developed a 
strategic plan for professional development of ALBE staff called A National 
Framework of Professional Development for Adult Literacy and Basic Education i^ersonnel. 

The evaluation was to examine the extent to which the aims and objectives of the 
Framework were achieved in the first year of implementation. (Impact of a FramaoorL 
for Professional Development of Adult Literacy and Basic Education Personnel, National 
Staff Development Committee for Vocational Education and Training, 1994, 
Chadstone Victoria.) 



Context 

Prior to the development of the Framework mentioned above little existed by the 
way of professional development for ALBE staff. There was a need for integrated 
structured professional development, with national applicability, and the consistent 
progressive development of skills and knowledge. 

In 1991 the TAFE NSDC developed the Framework and began its implementation. 
Part of the implementation strategy was a summative evaluation of the Framework 
to be conducted at a national level by an independent evaluator. 



The evaluation Question 

To what extent have the aims and objectives of the Natii .,ial Framework been 
achieved in the first year? 
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PURPOSE OF THE EVALUATION 




The evaluation purposes were: 



To identify any gaps in the Framework's objectives and delineate strengths and 
weaknesses of the professional development program. This is evaluation for 
program development (see Unit 2.4 of Module 2). 

To clarify, explain and interpret why any gaps exist. This is evaluation for 
program clarification (see Unit 2.5 of Module 2), 

To legitimate, account for and justify the resources used under the Framework. 
This is evaluation for program justification and accountability (see Unit 2.7 of 
Module 2). 

It was also noted that: 

'...the evaluation report will allow the TNSDC to make decisions about the 
continued improvement of the PD program for ALBE staff/ (p. 5) 

Hence the evaluation is also for program improvement (see Unit 2.6 of Module 2). 



WHO IS Involved? 

The stakeholders are NSDC personnel, people conducting PD for NSEXI, funding 
agencies, ALBE decision makers and ALBE staff. 

The evaluation was conducted by consultants Price Waterhouse Urwick. 



AUDIENCES 

Commonwealth Government, NSDC, ALBE decision makers and ALBE staff 



ERIC 



Methodology 

The evaluation was conducted in four stages. 

1 Analysis of existing information including the states' strategy documents and 
the formative evaluation questionnaires completed by PD participants at the 
end of training sessions. 

2 Focus groups conducted in all states and territories with an average of four 
participants comprising PD managers, presenters of PD programs and 
selected program participants. 

216 
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3 Interviews with ALBE managers in all states and territories. 



4 A national survey sent to a sample of approximately 250 ALBE teachers in all 
states and territories. 



FINDINGS 

The findings of the evaluation were summarised against the objectives of the 
Framework. 

Objective 1: 

To provide a nationally coordinated approach to the planning, development, 
implementation and evaluation of ALBE professional development programs over the 
next two years. 

The evaluators reported that while considerable progress has been made towards 
the achievement of the objective there was room for improvement. 

Objective 2: 

To ensure that adequate recruitment and induction processes are developed and 
implemented. 

The evaluators found that this was the 'least well handled' of all the objectives 
because the selection and induction of staff are not handled through a national 
body. 

Objectives 3 and 4: 

3 to promote opportunities for ALBE staff to develop additional competencies 

4 to increjise the competencies of ALBE personnel so that they can respond 
adequately to a broad range of learning situations or contexts. 

The finding was that these objectives were being achieved by the PD programs. 
However, there were some fundamental gaps, for example, the lack of a set of 
competency standards for ALBE staff and the fact that one third of the target 
population did not attend PD because of time or distance constraints. 

Objective 5: 

. To ensure that current practice is informed by research, and theoretical iffiplications are 
addressed through professional development opportunities. 

The evaluators concluded that there was little doubt that this objective was being 
achieved by the PD activities. 



ERIC 198 © NSDC/Commonwealth of Australia 1995 



PROFESSIONAL DEVELOPMENT FOR PROGRAM EVALUATION 



RECOMMENDATIONS 

The report makes 12 recommendations. 

• Some are designed to help program development suggesting priorities for action, 
for example: 

Develop more specific programs based on competencies, such as 
interpersonal skills, computer skills, etc. 

• Some are for program clarification suggesting how the program plan could be 
revised: 

Create national competency standards for ALBE staff. 

Standardise the format of the states' strategic and operational plans and 
define the performance indicators more precisely. 

• Others are directed at program improvement, for example: 

Develop a strategy to improve access of programs to sessional and part-time 
staff. 

Develop all appropriate programs in flexible delivery format. 

The different purposes overlap and are not separated in the report. In fact the 
report's findings and recommendations do not specifically refer to issues of program 
development and clarification, but concentrate on accountability issues and 
recommendations for the future. 



ISSUES ARISING 

The study illustrates how one evaluation can address all four common evaluation 
purposes outlined in Module 2. 

The study very clearly identifies the purposes of the evaluation in a section (p. 5) 
headed. Functions (Why are we evaluating?). It would have been useful if these 
purposes were re-addressed in the findings and recommendations. 

The use of four very different methods of information collection including 
interviews, focus groups and questionnaires improves the validity of the information 
collected (see Unit 2.6 of Module 4). 

The evaluation does not 'prove' that the program is effective (see Unit 5.3 of Module 
5), but the program is judged against its stated objectives. 
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Case Study 3 
Opportunity to Do Brilliantly: 

TAPE AND THE Challenge of Adult Literacy 
Provision in Australia 

WHAT IS BEING EVALUATED? 

Inside a larger research project contributing to national policy development two 
innovative adult literacy projects were evaluated. They were: 

1 The Council of Adult Education's literacy program in Melbourne in 1984 
which provided small group literacy tuition mostly located in the city centre, 

2 The Northern Districts Adult Literacy Group, known as the Glenroy Program, 
which, in 1984, used volunteer tutors for literacy teaching in the regional 
library and homes in the Broadmeadows area of Melbourne. 

The complete research report was published in 1987. (Grant, A. 1987, Opportunity to 
Do Brilliantly: TAPE and the challenge of adult literacy provision in Australia, Australian 
Government Publishing Service, Canberra) 



Context 

The study was undertaken at a time of 'unprecedented Commonwealth government 
initiatives' in the area of adult literacy and basic education. The author comments: 

'Never before has there been such groundswell and momentum at national, state 
and community levels, for righting the acute imbalance between the widespread 
need for adult literacy learning and the limited provision for actually meeting the 
demand.' (p. ix) 

The relevant Commonwealth department was seeking guidance on the development 
of policy about adult literacy and on making funding recommendations to 
Government, so it commissioned the research project as part of the process. 
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THE EVALUATION QUESTION 

The two evaluative case studies aim to describe what is distinctive about each 
program, what factors account for its successes and determine its appUcabihty 
elsewhere, and what constraints or limitations are evident. 



PURPOSE OF THE EVALUATION 

The funding body sought a research study focused on one aspect of adult literacy 
which would contribute to the development of policy in the adult literacy area. As a 
project 'concerned with the interface between needs and provision' it can be seen as 
concerned with program improvement, but there is a sense in which it is research for 
program development of the new adult literacy programs funded after 1985. 

The principal researcher conceived the research as an evaluation of two distinct 
models of adult Uteracy provision, reviewed in the wider context of recent 
developments in adult literacy. The evaluations were designed to describe: 

'what is most distinctive about each program, what factors account for its success 
and determine its applicability elsewhere, and what constraints or limitations are 
evident.' (p. 2) 

This is a style of evaluation for program improvement (see Unit 2.6 of Module 2), 
called responsive evaluation or naturalistic enquiry (see The Guide, p. 37 for more 
details). As a form of naturalistic enquiry it requires direct observation in the 
program setting and an openness to the issues which emerge. 

Because the research team included four program tutor-coordinators who collected 
much of the interview data, the evaluations have quite a participant evaluation or 
action research style about them (see Unit 6 of Module 2 and The Guide, pp. 37-38). 
In these models evaluation for improvement is closely integrated into program 
delivery (The Guide, pp. 38-39). 



WHO IS INVOLVED? 

The stakeholders were ALBE staff, course participants and ALBE management. 
The chief researcher and author of the report was Dr Audrey Grant of La Trobc 
University. The research team included four adult basic education 
tutors/coordinators who collected much of the interview data. 



Audiences 

Commonwealth government, funding agencies, ALBE managers and ALBE staff 
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METHODOLOGY 

Case study methods were used to make a descriptive evaluation of the tv^o models 
of adult literacy provision. 

The case studies were based on interviews with a cross-section of program staff, 
tutors and students. Interviews were open-ended, responsive to the interviewees 
and based on common topics rather than a detailed schedule. Group discussions 
were held with the two program coordinators and the research team. 

Records of program meetings, written observations and journals kept by 
participants, and coordinators' newsletters to tutors were also used as sources of 
information. 

The data was analysed by the research team identifying major themes and recurrent 
patterns in the data. Attention was also given to factors unique to a particular 
program and factors accounting for or restricting the success of the programs. 



FINDINGS 

Detailed findings were reported about the individual projects. 
Overall three major principles were identified: 

1 that the multiplicity of needs and contexts for adult literacy and basic 
education points to the necessity for diversity of program and plurality of 
provision; 

2 that a broad understanding of what becoming literate entails should inform 
all aspects of policy-making and implementation, program provision and 
evaluation research; 

3 that the participatory-learning-support model of program provision 
pioneered in adult literacy be safeguarded and replicated to facilitate greater 
equity of access in adult education. 

It is not possible here to summarise the 115 pages of the report describing the 
features of the two programs closely linked to the interview data. It can be said that 

♦ the report brings to life the programs described and 

♦ illuminates the issues they an^ grappling with. 

It is clear from the report that the interviews and the feedback provided to tutors and 
coordinators influenced the evolution of the programs and individual teaching and 
hence contributed to program improvement. Recommendation 1 indicates that the 
researchers themselves are very clear about the value of this form of evaluation for 
program improvement. 
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In the spirit of the evaluation some reflective quotes from one of the four tutor- 
coordinator evaluators might be illuminative. 

'If s really widened my vision of literacy I think the actual interviewing of the 

students made me really understand how. ..literacy affects their lives or their lives 
have affected their literacy.. .The research, because people have opened up and 
talked about it, has given me in a very short time a picture of different people 
and how they work. I think that's made me look differently at myself...' (p. 159) 



Recommendations 

Six major recommendations were made. 

1 The careful development of a continuing process of evaluation and 
participant research is an indispensable component throughout literacy 
provision: in ascertaining needs; in planning, developing and sustaining 
programs; in recruiting and training staff. Thereby that action-participant 
learning be the goal. 

2 That programs be encouraged to develop flexible structures and hence new 
initiatives which better care for students' growth and adult workers' potential 
contribution. That additional adult literacy field officers be appointed to 
facilitate this development. 

3 That policy implementation and allocation of funding promote devolution of 
responsibility so that decision making becomes increasingly the responsibility 
of experienced program participants. Such devolution must be preceded and 
accompanied by the development and funding of appropriate program 
support structures, staff development and training facilities, together with 
provision for participant-based evaluation and thereby for accountability. 

4 That, in the proposed development of existing and planned future services, 
the quality of staff recruitment, training and development be seen as crucial. 

5 That further research funding give priority to participant-evaluation, case- 
study based research. That this emphasis be promoted in commissioned 
research projects and in particular grants to institutions with an established 
commitment to teaching, postgraduate studies and qualitative research in 
language and literacy. 

6 The guidelines for the allocation of funding apply to the above principles and 
recommendations and recognise that the long-term commitment to students 
necessitates guaranteed funding over a three-year period. 
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PROFESSIONAL DEVELOPMENT FOR PROGRAM EVALUATION 



ISSUES ARISING 

The evaluation illustrates how responsive evaluations can contribute tc program 
improvement, particularly where the program staff also collect the evaluative 
information and hence get direct feedback on the results of their work (see Unit 2.6 
of Module 2). 

Such an approach also assists the implementation of evaluation findings (see Unit 2.7 
of Module 5) because the evaluator and the action-taker are one and the same 
person. 

The report shows how evaluations for program improvement, conducted on a small 
scale, can be used to contribute to principles and recommendations at the level of 
national policy-making and hence to program development (see Unit 2.4 of Module 
2). 

A limited case study methodology was adopted relying heavily on one data 
collection technique, interviews -a limitation forced by time and resource 
restrictions. This may have implications for the reliability, validity and fairness of the 
data (see Unit 2.6 of Module 4). 

The use of participant-evaluators results in excellent understanding of contextual 
issues (see Unit 1.7 of Module 1), but may make it more difficult for the evaluators to 
make unbiased judgments. 
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